Papers with image generation
PAI-Diffusion: Constructing and Serving a Family of Open Chinese Diffusion Models for Text-to-image Synthesis on the Cloud (2024.acl-demos)
Copied to clipboard
| Challenge: | Existing diffusion models fail to address the challenges of generating high-quality images from textual descriptions due to its large vocabulary size and complex character relationships. |
| Approach: | They propose a framework that integrates Chinese diffusion models with Alibaba Cloud's Platform for AI and enables the generation of contextually relevant images. |
| Outcome: | The proposed framework integrates with Alibaba Cloud’s Platform for AI, providing accessible and scalable solutions. |
Efficient Content-Based Sparse Attention with Routing Transformers (2021.tacl-1)
Copied to clipboard
| Challenge: | Self-attention suffers from quadratic computation and memory requirements with respect to sequence length . despite its effectiveness, self-attention models suffer from quadratic computation and a limited set of locations . |
| Approach: | They propose to learn dynamic sparse attention patterns that avoid allocating computation and memory to attend to content unrelated to the query of interest. |
| Outcome: | The proposed model outperforms similar sparse attention models on language modeling and image generation on Wikitext-103 . |
Generating Text through Adversarial Training Using Skip-Thought Vectors (N19-3)
Copied to clipboard
| Challenge: | Existing approaches to use word embeddings for text generation have been limited. |
| Approach: | They propose to use GANs with word embeddings to reproduce writing style in text . they use a sentence embeddable vector to model people's way of expression . |
| Outcome: | The proposed model outperforms baseline text generation networks across several metrics including BLEU-n, METEOR and ROUGE. |
SciSketch: An Open-source Framework for Automated Schematic Diagram Generation in Scientific Papers (2025.emnlp-demos)
Copied to clipboard
| Challenge: | SCISKETCH is an open-source framework that supports two automated workflows for schematic diagram generation using foundation models. |
| Approach: | They propose an open-source framework that supports two automated workflows for schematic diagram generation using foundation models. |
| Outcome: | The open-source framework outperforms several state-of-the-art foundation models in generating schematic diagrams for scientific papers. |
Generating Fine Details of Entity Interactions (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing text-to-image models excel at generating high-quality object-centric images from instructions, but lack of data for complex interactions. |
| Approach: | They propose a multimodal Large Language Models-generated dataset to benchmark and enhance interaction-rich images. |
| Outcome: | The proposed approach improves image quality and automatic and human evaluations show improvements. |
EasyGen: Easing Multimodal Generation with BiDiffuser and LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing multimodal models that depend on encoders like CLIP or ImageBind need ample amounts of training data to bridge modalities. |
| Approach: | They propose an efficient model that leverages bidirectional conditional diffusion model to foster more efficient modality interactions. |
| Outcome: | The proposed model is able to train a projection layer linking an LLM and an adapter to align the LLM’s text space with the bidirectional diffusion model. |
UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | e-commerce tasks such as multimodal retrieval and multimodal generation are largely ignored due to the diversity of the multimodal fashion domain. |
| Approach: | They propose a framework that integrates image generation with retrieval and text generation tasks. |
| Outcome: | The proposed framework outperforms state-of-the-art models across fashion tasks. |
ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work reveals that vision and language models struggle to comprehend fine grained distinctions in images. |
| Approach: | They propose a dataset to assess multimodal models' ability to match objects with their colors. |
| Outcome: | The proposed model performs well in visual questionanswering, text-to-image generation and word-order understanding tasks. |
TempViz: On the Evaluation of Temporal Knowledge in Text-to-Image Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing studies on temporal knowledge in text-to-image models have not explored how temporal phenomena are handled in text models. |
| Approach: | They propose a data set to holistically evaluate temporal knowledge in image generation using 7.9k prompts and more than 600 reference images. |
| Outcome: | The proposed model evaluates temporal knowledge in image generation using 7.9k prompts and more than 600 reference images. |
TC-Bench: Benchmarking Temporal Compositionality in Conditional Video Generation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing video generation models struggle to interpret compositional changes and synthesize components across different time steps. |
| Approach: | They propose a temporal compositionality benchmark that uses text prompts and ground truth videos to evaluate compositional changes in video. |
| Outcome: | The proposed benchmark can be used for text-to-video and image-to video generation. |
Interactive Text Generation (2023.emnlp-main)
Copied to clipboard
Felix Faltings, Michel Galley, Kianté Brantley, Baolin Peng, Weixin Cai, Yizhe Zhang, Jianfeng Gao, Bill Dolan
| Challenge: | Advances in generative modeling have made it possible to automatically generate high-quality texts, code, and images, but they can be unsatisfactory in many respects. |
| Approach: | They propose a task that allows training generation models interactively without the costs of involving real users. |
| Outcome: | The proposed model trains with Imitation Learning without the cost of involving real users and is superior to non-interactive models. |
A Cheaper and Better Diffusion Language Model with Soft-Masked Noise (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing diffusion models have limitations in modeling discrete data, e.g., languages . we present a novel diffusion model for language modeling inspired by linguistic features in languages based on iterative denoising . |
| Approach: | They propose a method that iteratively denoises and adds corruptions to the textual data through soft-masking to better noise it. |
| Outcome: | The proposed model achieves better generation quality and lower training cost than current models with better performance. |
ViPE: Visualise Pretty-much Everything (2023.emnlp-main)
Copied to clipboard
| Challenge: | Figure and non-literal expressions are deeply integrated in human communication . text-to-image models like Stable Diffusion struggle to depict non-figural expression . |
| Approach: | They propose a series of lightweight and robust language models that can be used to visualise non-literal expressions. |
| Outcome: | The proposed language models are more robust than existing models and can generate high-quality images. |
EditID: Training-Free Editable ID Customization for Text-to-Image Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-to-image models for customized IDs focus on ID consistency while neglecting editability. |
| Approach: | They propose a training-free approach to editable customized IDs based on the DiT architecture . EditID deconstructs existing text-to-image models into image generation branch and character feature branch . |
| Outcome: | The proposed solution achieves high-quality images with editable IDs while maintaining ID consistency. |
Polish Corpus of Annotated Descriptions of Images (L18-1)
Copied to clipboard
| Challenge: | a new dataset of image descriptions is presented in Polish . the dataset is too small for training a sophisticated language-vision system. |
| Approach: | They propose to use a Polish dataset to analyze image descriptions . the descriptions are morphosyntactically analysed and annotated by human annotators . |
| Outcome: | The proposed model learns about the inter-modal correspondences between language and vision. |
Continuous Language Generative Flow (2021.acl-long)
Copied to clipboard
| Challenge: | Recent years have witnessed various types of generative models for natural language generation (NLG), especially RNNs or transformers. |
| Approach: | They propose a flow-based language generation model that adapts flow-derived generative models to language generation via continuous input embeddings, adapted affine coupling structures, and a novel architecture for autoregressive text generation. |
| Outcome: | The proposed model improves on QG and NMT and improves performance over baselines on SQuAD and TVQA and NML16. |
RefVNLI: Towards Scalable Evaluation of Subject-driven Text-to-image Generation (2025.findings-emnlp)
Copied to clipboard
Aviv Slobodkin, Hagai Taitelbaum, Yonatan Bitton, Brian Gordon, Michal Sokolik, Nitzan Bitton Guetta, Almog Gueta, Royi Rassin, Dani Lischinski, Idan Szpektor
| Challenge: | Existing methods assess only one aspect of the task, misalign with human judgments or rely on costly API-based evaluation. |
| Approach: | RefVNLI evaluates textual alignment and subject preservation in a single run. |
| Outcome: | RefVNLI outperforms or matches existing baselines across multiple benchmarks and subject categories. |
Bayesian Optimization for Controlled Image Editing via LLMs (2025.findings-acl)
Copied to clipboard
Chengkun Cai, Haoliang Liu, Xu Zhao, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, John Lee, Jenq-Neng Hwang, Lei Li
| Challenge: | achieving precise control over generated content and maintaining semantic consistency remain significant limitations, particularly concerning grounding techniques and the necessity for model fine-tuning. |
| Approach: | They propose an off-the-shelf approach that integrates Large Language Models with Bayesian Optimization to facilitate precise and user-friendly image editing. |
| Outcome: | The proposed approach outperforms existing methods in editing accuracy and semantic preservation, as validated using different LLMs including Claude3 and GPT-4. |
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to build a strong multilingual multimodal representation model are lacking in good-quality text-image pairs. |
| Approach: | They propose a method to build a strong multilingual multimodal representation model using English text-image pairs instead of a model from scratch. |
| Outcome: | The proposed model outperforms the original CLIP model on multilingual multimodal benchmarks. |
Character-centric Story Visualization via Visual Planning and Token Alignment (2022.emnlp-main)
Copied to clipboard
| Challenge: | Story visualization is a task that requires machines to understand long text inputs and produce a globally consistent image sequence that illustrates the contents of the story. |
| Approach: | They propose to augment VQ-VAE with a text-to-visual-token (transformer) architecture to enable multiple image generation based on a complete story. |
| Outcome: | The proposed method excels at preserving characters and produces higher quality image sequences compared with baselines. |
Chain-of-Jailbreak Attack for Image Generation Models via Step by Step Editing (2025.findings-acl)
Copied to clipboard
Wenxuan Wang, Kuiyi Gao, Youliang Yuan, Jen-tse Huang, Qiuzhi Liu, Shuai Wang, Wenxiang Jiao, Zhaopeng Tu
| Challenge: | Text-based image generation models, such as Stable Diffusion and DALL-E 3, hold significant potential in content creation and publishing workflows . however, considerable efforts are being made to prevent the generation of harmful content, such abusive, violent, or pornographic material. |
| Approach: | They propose a chain-of-jailbreak method which decomposes malicious queries into multiple sub-queries and iteratively edits images based on these sub-questions. |
| Outcome: | The proposed method can bypass safeguards of image generation models for over 60% cases, significantly outperforms other jailbreaking methods (14%) |
Exploring Precision and Recall to assess the quality and diversity of LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for large language models are limited to specific tasks, but they are now widely available for a wide range of tasks. |
| Approach: | They propose a framework for large language models such as Llama-2 and Mistral that imports precision and recall metrics from image generation to text generation. |
| Outcome: | The proposed framework allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora. |
Poetry in Pixels: Prompt Tuning for Poem Image Generation via Diffusion Models (2025.coling-main)
Copied to clipboard
Sofia Jamil, Bollampalli Areen Reddy, Raghvendra Kumar, Sriparna Saha, Joseph K. J, Koustava Goswami
| Challenge: | Poems are a distinct form of literature, with meanings that transcend beyond the literal words. |
| Approach: | They propose a framework to generate images that visually represent the meanings of poems using prompt tuning and a PoeKey algorithm to extract emotions, visual elements, and themes from poems. |
| Outcome: | The proposed framework generates images that visually represent the meanings of poems and their images. |
Repairing Catastrophic-Neglect in Text-to-Image Diffusion Models via Attention-Guided Feature Enhancement (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Text-to-Image Diffusion models generate high-quality images from textual descriptions, but they often produce images that do not fully align with the input prompts, resulting in semantic inconsistencies. |
| Approach: | They propose an automated repair approach to address catastrophic-neglect in T2I DMs. |
| Outcome: | The proposed model achieves 10.1%-16.3% higher Correct Rate in image generation compared to baselines. |
Generating Contextual Images for Long-Form Text (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in Text-to-Image models require short prompts that describe both the content and style of the target image. |
| Approach: | They propose to use Large Language Models (LLMs) and Text-to-Image Models to synthesize relevant visual imagery from generic long-form text. |
| Outcome: | The proposed models can generate high-quality images from short prompts that describe both the content and style of the target image. |
Textual Aesthetics in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on image aesthetics have focused on content correctness and helpfulness of responses. |
| Approach: | They propose a textual aesthetics-powered fine-tuning method that leverages textual visual aesthetics without compromising content correctness. |
| Outcome: | The proposed method improves aesthetic scores and performs well on general evaluation datasets. |
FineRAG: Fine-grained Retrieval-Augmented Text-to-Image Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Recent advances in text-to-image generation still exhibit limitations in terms of knowledge access. |
| Approach: | They propose a fine-grained retrieval-augmented image generation model that breaks down the retrieval task into four critical stages: query decomposition, candidate selection, retrieval augmented diffusion, and self-reflection. |
| Outcome: | The proposed method significantly reduces noise associated with retrieval-augmented image generation and performs better in complex, open-world scenarios. |
The Revolution of Multimodal Large Language Models: A Survey (2024.findings-acl)
Copied to clipboard
Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara
| Challenge: | Recent advances in large language models have led to the development of multimodal large language model. |
| Approach: | They present a review of recent visual-based Large Language Models and analyze their architectures and alignment strategies. |
| Outcome: | The proposed models can integrate visual and textual modalities while providing a dialogue-based interface and instruction-following capabilities. |
Do It Yourself (DIY): Modifying Images for Poems in a Zero-Shot Setting Using Weighted Prompt Manipulation (2025.emnlp-main)
Copied to clipboard
| Challenge: | a novel method to enhance imagery in poetic language is proposed . weighted prompt manipulation is a new approach to enhance poetry images . current diffusion models struggle to interpret metaphorical language, symbolism, and nuanced themes. |
| Approach: | They propose a weighted prompt manipulation technique that modifies attention weights and text embeddings within diffusion models to enhance or suppress specific words' influence in the final generated image. |
| Outcome: | The proposed technique enhances or suppresses the influence of specific words in the final generated image, leading to semantically richer and more contextually accurate visualizations. |
AcT2I: Evaluating and Improving Action Depiction in Text-to-Image Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-Image (T2I) models have been successful in generating images from textual descriptions, but they struggle to capture nuanced and implicit attributes inherent in action depiction. |
| Approach: | They propose a benchmark to evaluate the performance of T2I models in generating images from action-centric prompts. |
| Outcome: | The proposed model achieves an increase of 72% on AcT2I. |
Misalignment Attack on Text-to-Image Models via Text Embedding Optimization and Inversion (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Text embedding is a key component of modern NLP models but also poses additional risks. |
| Approach: | They propose a framework that optimizes embeddings and inverts them to obtain misaligned prompts. |
| Outcome: | The proposed framework exploits the continuity and distribution characteristics of text embeddings to obtain misaligned prompts of discrete tokens. |
Beyond Content: How Grammatical Gender Shapes Visual Representation in Text-to-Image Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | grammatical gender significantly influences image generation in text-to-image models . masculine grammatikal markers increase male representation to 73% on average . feminine grammatological markers increase female representation to 38% . |
| Approach: | They propose a cross-linguistic benchmark examining words where grammatical gender contradicts stereotypical gender associations. |
| Outcome: | The proposed benchmark examines words where grammatical gender contradicts stereotypical gender associations. |
Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing autoregressive models have shown superior performance and efficiency in image generation, but are constrained by high computational costs and prolonged training times in video generation. |
| Approach: | They propose a Local Optimization method which optimizes tokens within localized windows while leveraging contextual information to reduce error propagation. |
| Outcome: | The proposed method achieves superior performance to the baseline while halving the training cost without sacrificing quality. |
MENTOR: Efficient Autoregressive Image Generation with Balanced Multimodal Control (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent text-to-image models achieve impressive visual quality but still face challenges in precise controllability, balancing multimodal inputs, and high training cost for multimodal image generation. |
| Approach: | They propose an autoregressive framework with a two-stage training paradigm for controllable multimodal image generation. |
| Outcome: | Extensive experiments on DreamBench++ and DreamBech show that the proposed framework achieves a strong balance between textual and visual guidance for controllable image generation. |
Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work focused on improving alignment by refining the diffusion process, ignoring the role of the text encoder, which guides the diffusion. |
| Approach: | They investigate how semantic information is distributed across token representations in text-to-image prompts by patching techniques to uncover encoding patterns. |
| Outcome: | The proposed model can improve alignment and generation quality by modifying the diffusion stage and the cross-attention mechanism. |
When Cultures Meet: Multicultural Text-to-Image Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | a new task to evaluate text-to-image generation models for multicultural scenes is unexplored. |
| Approach: | They propose a benchmark task to evaluate text-to-image models in multicultural settings . they use a dataset of 9,000 images spanning five countries, three age groups, two genders, 25 historical landmarks, and five languages to analyze behavior . |
| Outcome: | The proposed benchmark analyzes the behavior of state-of-the-art models across multiple dimensions including alignment, image quality, aesthetics, knowledge, and fairness. |
ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Services (2026.acl-long)
Copied to clipboard
Fengxian Ji, Jingpu Yang, Zirui Song, Lang Gao, Junhong Liang, Zhenhao Chen, Jinghui Zhang, Xiuying Chen
| Challenge: | Recent image generation and editing models demonstrate robust adherence to instructions and high visual quality on academic benchmarks. |
| Approach: | They propose a benchmark that correlates image outputs with economic value in commercial design projects. |
| Outcome: | ServImage benchmarks show image generation models perform well on academic benchmarks but are uncertain on commercial projects. |